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I. INTRODUCTION 


A. AREA OF RESEARCH 


The purpose of this thesis is to propose a new learning medium that takes 
advantage of the latest developments in computer-based training (CBT) and the World 
Wide Web (WWW) as an innovative mode for delivering education. The research focus 
is the design of a general framework for developing integrated Web-based CBT tools that 
combine hypertext technology, computer-based training functionalities and interactive 


multimedia to enhance learning effectiveness. 


B. RESEARCH OUESTIONS 


e What are the design considerations and implementation strategies for 
combining hypertext, multimedia case studies and group decision support 
(GDSS) software? 

e What is the ideal structure of a web based training tool? 

e What is the best way to implement a multimedia case study? 


e How can DSS software be used in an interactive training tool? 


e What are the difficulties associated with creating a web based tool as opposed 
to a stand-alone application? 


E. BENEFITS OF STUDY 


There is a need for continuous education and training; most specifically training 
in real-time for effective information usage. This study contributes to meeting this need 
through creation of a framework that can be used to analyze, design, implement and 


evaluate a web-based training presentation. 


D. METHODOLOGY 


To identify the reguired elements for designing and developing a web-based 
training program that integrates hypertext, multimedia case studies and group decision 
support system (GDSS) software, we conducted a survey and analysis of CBT 
development issues. We also performed a comparison of CBT authoring tools to 
determine which tool provided the best features and capabilities for creating training 
programs and deploying them on the web. 

To demonstrate the feasibility of implementing an integrated training program on 
the Internet, we used the ELECTRE 1 Model from Co-oP to develop a training prototype 
using Macromedia Director 6 presentation and authoring software. Co-oP is a Group 
Decision Support System for Cooperative Multiple Criteria Group Decision Making, 
created by Dr. Tung X. Bui of the Naval Postgraduate School, Monterey, California. 

In order to capture vital user feedback for the enhancement of the prototype, we 


created an on-line evaluation form using NetObjects Fusion 2.0 and Microsoft Access 97. 
E. RESEARCH OUTPUT 


A working prototype of a Web-based CBT was created with Macromedia Director 
6.0. The product is available for viewing at [http://www.cimnet.nps.navy.mil/coop). 

The software training module is provided "as is." All guarantees either expressed 
or implied are disclaimed as to the software and its quality, performance, or fitness for 


any particular purpose. 


F. ORGANIZATION OF THE STUDY 


Chapter II is a literature review of CBT. A brief history of CBT is presented. 
Current trends in CBT delivery such as multimedia, authoring tools and Internet/WWW 
platform deployment are discussed. 

Systems analysis and design procedures employed in developing the training 
module are discussed in Chapter III. The framework for the development methodology is 
derived from the Instructional Systems Development (ISD) model discussed in Chapter 
II. Design considerations and trade-off issues are also discussed. 

Chapter IV discusses the operating procedures and system functionality for the 


training module. This chapter includes a description of: 


e minimum system requirements for accessing and viewing the training module 
e ideal system features for optimizing the training module’s access and viewing 


e approximate download times for various connection rates 


Chapter V concludes the study by summarizing the research, discussing lessons 


learned and providing recommendations for future research. 





II. LITERATURE REVIEW 


A. HISTORICAL PERSPECTIVE OE CBT 


Despite having gained significant popularity in recent years, CBT has been 
around for several decades. Maul and Spotts (1993) trace CBT's origins back to the 
1920s with Pressey's invention of a teaching machine that could “produce measurable 
amounts of learning in students.” Much of the credit for the instructional principles 
behind CBT courseware is attributed to B.F. Skinner (Hassett and Duke, 1986). In the 
1950's Skinner developed a teaching machine which according to Hassett and Dukes: 

...presented instruction and questions on a paper tape which the student 

moved to see a new sentences at a time. The user wrote answers on a 

second tape, then uncovered a hidden portion of the teaching tape to see 

whether the answer was correct. 

Easton (1997) defines CBT as “the use of computers and multimedia technology 
for training in a way that promotes student interest and motivation.” Much of the 
literature reviewed contained numerous acronyms describing the use of computers in 
education and training. Other CBT related terminology described by Bixler and Bergman 
(1997) includes the following: 

e CAI — Computer-Assisted Instruction, and CBI — Computer-Based 


Instruction; delivering instruction through the medium of a computer; 
these are alternative terms for computer-based training 


e CAL — Computer-Assisted Learning; using a computer as an adjunct 
tool to aid in a traditional learning situation, like classroom training. 
The computer is a device to assist the instructor during the training 
process; like a blackboard or handouts 

e CAT — Computer-Assisted Testing. Assessing an individual through 

the medium of a computer. Individuals take the test at the computer, 
and the computer records and scores the test. CAT is imbedded in 
most computer-based training 

e CMI — Computer-Managed Instruction. The computer is used to 

register students, assign training schedules, generate and score tests, 
and produce reports on student progress and utilization of training 
resources (Kearsley, 1983) 

Fletcher and Rockway (1986) date the history of computer based instruction 
(CBI) as beginning in the mid-1950’s. Much of the initial CBI research and development 
was either performed by or supported by the military. The Air Force is credited with 
being the first military service to begin developing CBI (Fletcher & Rockway, 1986). The 
Air Force’s Semi-Automatic Ground Environment (SAGE), was the first “computer- 
based, on-the-job training system” (Fletcher & Rockway, 1986). 

The Navy is regarded as the slowest military service to begin participation in CBT 
research and development (Fletcher & Rockway, 1986). Despite its slow entry into the 
CBT field, the Navy is recognized for its success at the U.S. Naval Academy, which was 
“among the first institutions of higher learning to offer an entire course by computer” 
(Fletcher & Rockway, 1986). 

The most widely recognized CBT initiative involving Department of Defense 


agencies, was the Programmed Logic for Automatic Teaching Operations (PLATO) 


Project. The initial development period for the PLATO Project occurred from 


approximately 1959 to 1965. During this period, researchers received funding support 
from the Army, Navy and Air Force. Fletcher and Rockway (1986) describe the initial 
product, PLATO I, as “a single-station computer-based teaching machine using the 
ILLIAC (with 1K 40-bit words of primary memory, a 10,000-word drum, and an add 
time of 75 microseconds) as the central processor.” The PLATO project successfully 
provided the first programming environment for computer-based training. 

Nicholson (1997) credits the evolution of the microcomputer in the late 1970’s as 
the springboard for the principle of developing programs to provide electronic training 
which supplements or replaces standard training manuals. Programs developed during 
this time were consistent with one of two fundamental layouts—linear programs and 
branching programs (Nicholson, 1997). The linear program involved “a sequence of very 
small learning steps punctuated by a test question.” This method provided immediate 
feedback to the student as to whether the question had been answered correctly. One of 
the most widely recognized linear programs is “Exploring the IBM Personal Computer”. 
The program was distributed as an introductory disk, which was shipped with each new 
IBM personal computer (Hassett and Dukes, 1986). In branching programs, students were 
provided with one of several instruction paths to help lead them to correctly answering 
questions. The program would not allow the learner to continue to the next step of the 
program without the answer given being correct. 

Modern technological advances have resulted in the creation of media-rich 
training programs, which combines text, graphics, animation, video and digital sound. 


Multimedia enables CBT courseware developers to “create a complete multi-sensory 


learning program, allowing students to interact with the material, and to learn according 
to their own needs, pace and learning styles.” (Easton, 1997) These multi-sensory 
systems aid in the greater retention of new knowledge according to Bixler and Bergman 
(1997), who observed that “on average, people remember: 10% of what they read; 20% 
of what they hear; 30% of what they see; and 50% of what they hear and see.” 

CBT program designs incorporate one or a combination of the following 


techniques (Bixler & Bergman, 1997; Kearsley 1983): 


e Tutorial — useful for teaching factual information, simple discrimination, rules 
and simple application of rules 


e Drill and Practice — used after initial instruction to aid in mastery of new skills 
or information taught 


e Training Games — used as a supplement to other instruction. Training games 
provide users with motivating opportunities to practice new skills or 
information taught 


e Simulation — used primarily to practice skills in situations where real-world 
application is too expensive or dangerous 


e Problem Solving — helps student develop skills in logic, problem solving, and 


following directions, and is generally used to augment higher order thinking 
skills 


e Demonstration/Presentation — best used to support the introduction of new 
information 


B. CURRENT TRENDS IN CBT DELIVERY 


1. Determining the Feasibility of CBT Use 


Kearsley (1983) stated, “almost all CBT applications are driven by the need to 
improve the efficiency or effectiveness of a particular training system or situation.” This 
assertion continues to hold true across the wide spectrum of training and educational 
organizations. In an era of military and corporate downsizing, training specialists are 
continually searching for more efficient ways to spend limited training dollars. Allen 
(1996) originally highlighted challenges that he believed would have a major impact on 


training and instruction in the early 1980's. These challenges include the following: 


e The high cost of energy 

e Generally inefficient use of instructors due to student overload 

e Changing social trends 

e The high cost of training in multiple locations 

e Too wide a range of learner abilities and expectations 

e The increasing cost of downtime required by traditional methods 
e Increased turn-over and worker/student mobility 


e The inability of traditional training modes to present instruction 
consistently across learner groups 


e General information overload 


Many of these same challenges are being encountered by training and education 


Organizations today. These are some of the issues that form the basis for exploring CBT 


as a training option (Allan 1993; Rand, 1996; Gallagher & Olbeter, 1997). Montague 
(1984) studied the use of CBT systems for training in the U.S. Navy and identified the 


following reasons for using CBT, although they are not unique to the U.S. Navy: 


e Provide students direct contact with tasks to be learned via simulations 
e Reduce the variability of instruction 

e Improve security and quality of achievement testing 

e Reduce requirements for presence of instructors 

e Make training available at anytime to match student availability 

e Provide training at remote site where instructors are unavailable 

e Reduce training time 

e Convenience 

e Reduce travel requirements and per diem expenses for training 


e Reduce development time for teaching materials and revisions 


These recurring themes seem to provide adequate rationale for implementing 
CBT. However, CBT may not be an appropriate solution for some training Situations. 
Extreme care and consideration must be exercised when determining when, where and 
how to use CBT. Due to the potentially high investment cost of CBT, organizations must 
perform assessments to determine: (1) the feasibility of incorporating CBT into their 
training strategy; (2) the type of CBT courseware that best fits their training needs and (3) 
the return on investment or cost-benefit analysis of CBT. Kearsley (1983), proposed a 


“CBT Benefits Checklist” (Table 2.1), to aid in assessing a training situation and the 


potential benefit to be derived from applying CBT technology. The checklist is not 


9 


designed to be a conclusive evaluation tool. A majority of “yes” answers should be 
viewed as a situation where CBT may successfully address the organization's training 
needs and warrants further investigation. A majority of “no” answers is indicative of a 


situation where CBT would most likely not fit the organization’s training needs and 


alternative methods should be considered. 
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CBT BENEFITS CHECKLIST 


Increased Control 

Are existing materials poorly used? 

Are existing training programs taught inconsistently? 
Is standardization of training important? 

Is detailed tracking of learning needed? 


Reduced Resource Requirements 
Is decentralized (field-based) training possible? 


Is higher student throughput desired? 
Is a higher student-to-instructor ration desired? 
Is expensive equipment needed for training? 


Individualization 

Is there considerable variation in student backgrounds? 
Is there considerable variation in student abilities? 

Is there likely to be considerable student variation in 
terms of learning progress? 

Does the instruction have to stand alone 

(e.g., self-study)? 


Timeliness and Availability 

Is it necessary to provide training to many students 
as quickly as possible? 

Is it desirable to provide training “on demand” 

i.e., Whenever and wherever a student needs it? 

Is there a problem with students forgetting due to 
premature training? 

Is there a problem with a shortage of qualified 
instructors? 


Reduced Training Time 

Would time savings in training be worthwhile? 
Can the training system and organization be 
changed to capitalize on time savings? 


(Continued on next page) 
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6. Improved Job Performance 
Is the guality of job performance a critical training 


concern? 
Are there job performance problems that improved 
training can address? 


7. Convenience 
Do employees already use a computer system for 
their jobs? 
Could CBT be integrated into existing jobs or 
equipment? 


8. Change Agent 
Is there a need for new training approaches or 


methods? 
Could CBT lead to improved personal or 
organizational productivity? 


2 Increased Leaming Satisfaction 
Is the attrition or failure rate high? 


Is there a problem with student motivation? 


10. Reduced Development Time 
Is the large-scale development of training materials 


and program involved? 

Is immediate revision/update of training material 
important? 

Is instruction developed in terms of a competency- 
based framework? 


Yes 
[|] 
[ ] 
[ ] 
[ ] 
[ ] 
le 
[ ] 
al 
lumel 
= 
lid 


Table 2.1 CBT Benefits Checklist (Kearsley, 1983) 
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No 
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The purpose of a feasibility study is to investigate and assess the degree of risk 
associated with using CBT. According to Kearsley (1983), there are four major types of 
feasibility that need to be examined: 

e Instructional — deals with the suitability of CBT for achieving the 

instructional objectives of the training 


e Organizational — focuses on social, policy or political issues that may 
affect the use of CBT 


e Technical - availability of CBT features or capabilities needed 


e Economic — concerns the affordability of CBT costs involved 


Kearsley further contends that a CBT application should be successful in all four 
areas in order to be successful. 

Adams (1993) offers a “decision aid” to help determine whether to use CBT in a 
corporate environment. The goal is to provide an evaluator with “a structured way to look 
at issues involving the learners and the corporate climate.” Adams also advises that the 
aid should not be viewed as a substitute for detailed cost-benefits analysis. The decision 
aid should be used as a supplement to a cost-benefit analysis. 

Much attention has been given to measuring the return on investment (ROI) for 
information technology acquisitions in both the public and private sector. In assessing the 
return on investment of CBT, Allen (1996) examines the potential results of CBT from 
two main factors: cost savings and performance improvement. Allen develops a cost 


savings model for CBT by illustrating savings in the following areas: 
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e Wages spent on training and opportunity costs 
e Travel and entertainment costs for learners 

e Travel and entertainment costs for trainers 

e Development costs 


e Delivery system costs 


In summarizing the research regarding CBT’s impact on worker performance, 
Allen (1996) concluded that “people learn better with multimedia training; they 
remember what they learn more accurately and longer, and they are better able to use 


what they learn to improve their performance.” 


2. Cost Issues 


CBT programs generally come in one of three types: (1) off-the-shelf; (2) 
customizable programs; and (3) programs built from scratch and fully customized to 
company specifications (Bixler & Bergman 1997). Off-the-shelf training is the cheapest 
and least flexible type of CBT because the content is fixed and cannot be changed. Off- 
the-shelf CBT is considered appropriate for training situations where the skills to be 
taught are relatively generic. Customizable CBT allows for the addition of organization 
specific content and is best suited for teaching skills that are relatively unique. 
Customized training is the most expensive type of CBT to develop and implement. As a 
result, this type of CBT is recommended for situations where the skills to be taught are 
unique, there is no existing CBT to teach the skills and there is a requirement to train a 


large number of people. 
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A variety of approaches have been used for estimating the cost of purchasing or 
developing CBT. Bixler and Bergman (1997) use a combination of development and 
delivery costs. These costs are dependent upon the customizability of the CBT program 
and the instructional format of the training. Tables 2.2 - 2.4 illustrate the development 


costs, delivery costs and budget matrix derived by Bixler and Bergman. 


DEVELOPMENT COSTS 
Computer $100 - $300 | $100 - $5,000 $100 - $2,000 
Program/software 
Customization or $1,280 $30,720 - $46,080 
Development 
installation 
TOTAL $740 - $940 | $2,020 - $6,920 | $31,460 — $48,720 


Table 2.2 Current CBT Development Costs (Bixler & Bergman, 1997) 




















DELIVERY COSTS 
Independent Study Off-site Classes 


M a gpm 
Instructional Support $1,280 $768 $512 


TOTAL $1,280 $1,536 Slee 521,152 


Table 2.3 Current CBT Delivery Costs (Bixler & Bergman, 1997) 











BUDGET MATRIX 


Independent | On-Site Off-Site 
Classes Classes 
$1,280 $1,536 $1,752 — 
| [mus 
Off-the-shelf $740 — $2,020 — $2,276 — 
| | ao | sew | 
Customizable | $2,020- $3,300 — $3,556 — 
ia bed $6,920 $8,200 $8,456 ee 
$31,460- | $32,740 - zu 


$48,720 $50,000 
Table 2.4 Current CBT Budget Matrix (Bixler & Bergman, 1997) 







Development Costs 























$50,256 
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Military instruction and training reguirements tend to be unigue therefore most 
CBT programs used within the military must be customized. Cost figures for military 
skills related CBT were obtained from the Chief of Naval Operations Office of Training 
Technology (OTT), paper on Interactive Courseware (ICW). Costs are estimated 
according to the category of ICW presentation. The following table provides a summary 


of ICW presentation categories and estimated average cost per hour of instruction. (OTT, 


Hour of Instruction 
| 
text 
| 
minor text 
3 High level simulation 


Table 2.5 Cost Estimate for Interactive Courseware (ICW) (OTT, 1996) 


1996) 





Category Number 













3. Hypertext-based Instruction 


In researching instructional methodologies and adult learning, Montague and 
Knirk (1997) determined that hypertext-based instruction is appropriate for conveying 
complex content. Traditional instructional methods tend to be linear in form and more 
suitable for well-structured subject matter. Hypertext provides for a non-sequential, 
random access method of obtaining information. (Montague & Knirk, 1997) 

The structure of hypertext, namely nodes and links, presents a unique architecture 


for the development of instructional materials. Balsubramanian (1994) defined nodes as 
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“concepts” and links as “relationships.” Nodes can contain text, graphics, video, audio, 
animation and images. Links are responsible for establishing connectivity between nodes; 
thereby connecting related concepts. The basic features of a hypertext system are 


(Balasubramanian, 1994): 


A Graphical User Interface, with the help of browsers and overview 
diagrams, helps the user to navigate through large amounts of information 
by activating links and reading the contents of nodes 


e An authoring system with tools to create and manage nodes (of multiple 
media) and links 


e Traditional information retrieval (IR) mechanisms such as keyword 
searches, author searches etc. 


e A hypermedia engine to manage information about nodes and links 
e A storage system which can be a file system or a knowledge base or a 
relational database management system or an object-oriented database 
management system 
Implementing a hypertext system on the World Wide Web creates an instructional 
environment where content can be developed, modified and maintained in a timely 
manner. These features are particularly attractive when there is a need to distribute 
current and rapidly changing information and instruction (Hites & Ewing, 1997). The 
cross-platform capability of the web guarantees users are able to access materials with a 


variety of systems and browsers. 
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4. Multimedia as a Training Tool 


According to the 1996 Computer-Based Training Report (Kemske, 1996), 73% of 
1,024 survey respondents estimated multimedia was used to some degree in their 
organization’s courseware. Respondents were also asked to compare multimedia to 
traditional CBT based on five criteria: 

e knowledge outcomes 

e performance outcomes 

e overall return on investment 

e retention 

e instruction completion time 

The most significant findings occurred in the area of knowledge outcomes, where 
79% of the respondents felt multimedia was better than traditional CBT. 

In a 1996 survey of 146 Fortune 1,000 companies by TRAINING and OmniTech 
Consulting Group of Chicago, multimedia as a training delivery system accounted for an 
estimated 16% of the total training hours delivered to employees (Training, 1996). For 
the purposes of the survey, the investigators defined multi-media based training (MBT) 
as “...an interactive learning experience incorporating the use of either CD-ROM or 
World Wide Web technology (via the Internet or, more often, an internal company 
“intranet” using the WWW protocols).” 

To further refine the survey results, respondents were grouped into three 


classifications based on the level of multimedia use within their organization. The 
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classification groups were light users, moderate users, and heavy users. Results for each 
classification are as follows: 
e Light users — are companies that deliver less than10 percent of total 
training hours via multimedia. Light users comprise 36 percent of all 


survey respondents 


e Moderate users — deliver 10 percent to 25 percent of total training 
hours via multimedia. They comprise 42 percent of respondents 


e Heavy users — comprising 17 percent of respondents, deliver more 
than 25 percent of all training hours via multimedia 


e The remaining 5 percent of respondents declined to estimate a figure 


In addition, the results indicated that software/basic computer skills remains the 
most freguently taught subject via CBT. Table 2.6 provides a breakdown of other training 


topics taught (Training, 1996). 


a 










Interpersonal Skill Training 


Job Responsibility Training 
(Basic, noncomputer-related) 









Customer Service Training 
Sales/Marketing Training 
Corporate Culture Training 


Advanced Computer Training 


Table 2.6 Training Topics Taught via CBT (Training, 1996) 


5. Internet-based Training (IBT) and Web-based Training (WBT) 


The Internet is becoming an increasingly popular alternative for delivering 
training (Appleton, 1997; McManus 1995; McGarvey, 1996; Schneider, 1994). Training 
delivered over the web provides for easy distribution of knowledge and resources 
regionally, globally, or within an enterprise. McManus (1995) provides a description of 


the Internet as a delivery system that also adequately describes the World Wide Web. 
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The Internet can deliver video, but not as guickly as videotape, television 
or CD-ROM. It can carry real time personal interaction, but not as well as 
telephone or video conferencing. It can display textual information, but 
not as usefully as a book or magazine. Why then should the Internet ever 
be used. The Net has two real advantages over other media. It combines 
advantages of other media so that it conveys video and sound better than a 
book, is more interactive than a videotape and, unlike CD-ROM, it can 
link people from around the world cheaply. The second advantage, and 
one that is often overlooked when discussing the Internet as a delivery 
system, 1s that it can also be a content provider. The Internet is, arguably, 
the largest and most diverse information resource in the world today. It is 
possible to incorporate a wealth of information available on the Net in 
your design. ...This sort of immediate access to information and resources 
can not be found with any other medium. (McManus, 1995) 

In comparing WBT to traditional implementations of CBT, Kilby (1994) outlines 

the following advantages and disadvantages of WBT: 


Advantages: 


e Easy delivery of training to users 

e Instant multi-platform capabilities (Windows, Mac, Unix) 

e Easy updating of content 

e Shift from costly programming to interface and content design 

e Quicker turnaround of finished product 

e Requires less technical support 

e Billing options by user ID, number of accesses, date/time of access 
e Access is controllable 


e Options for installations on private networks for security or greater 
bandwidth 


e Options to link with other training systems 


23 


e Multitasking capability suitable for electronic performance support 
systems (EPSS) 


e Vast, untapped market for training 


Disadvantages: 
e Limited formatting of content in current browsers 


e Bandwidth/browser limitations may restrict instructional 
methodologies 


e Limited bandwidth means slower performance for sound, video, and 
intense graphics 


e Someone must provide server access, control usage, bill users 


Most of the disadvantages of WBT are attributable to techonolgical limitations. 
As new advancements in browser capabilities and solutions to bandwidth limitation 


problems are discovered, WBT’s popularity as a training alternative, will increase. 


6. Motion Video 


The use of video in CBT programs and on the Web is becoming more common. 
However, video reguires enormous storage space and processing power (Joralmon, 
1997). One frame of video reguires approximately one megabyte of storage space. Full 
video uses 30 frames of video per second (Joralmon, 1997). The Moving Picture Experts 
Group (MPEG) standard developed by the International Standards Organization and the 
Intemational Electrotechnical Commission, is becoming a common compression 
standard for CBT and Web applications (Joralmon, 1997). The three MPEG standards 


compress video into files that are smaller and more suitable for use with current computer 


24 


resources. Specifically, “MPEG-1 standard compresses video into files with playback 


data rates supported by current desktop computers.” (Joralmon, 1997) 


C. INSTRUCTIONAL DESIGN ISSUES 


Regardless of the type of courseware developed, CBT, IBT or WBT, the 
courseware must be based upon some model of instruction and learning (Kearsley, 1983; 
McManus, 1995; Schneider, 1994). One of the greatest dangers associated with current 
advancements in CBT development and delivery systems is that the quality of the courses 
will decrease as responsibility for course development moves away from the training and 
instructional design specialist and more towards the information systems department 
(Fritz, 1993; Hefner, 1996). 

Instructional Systems Development (ISD) refers to “the systematic approach to 
design and development of training materials and systems.” (Kearsley, 1983) The five 
major phases of ISD are shown in Figure 2.1. ISD procedures help to minimize the 
problems most often attributed to training program failures. It represents the foundation 


for designing and developing effective CBT courseware. (Kearsley, 1983) 
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DESIGN +——— | DEVELOPMENT 


ANALYSIS Ñ / IMPLEMENTATION 
N [sees EVALUATION mamm | 7 





Figure 2.1 Major Phases of ISD (Kearsley, 1983) 

McManus (1996) uses Cognitive Flexibility Theory (Spiro, Feltovich, Jacobson, 
and Coulson, 1991) as the foundation for advocating use of the Hypermedia Design 
Model (HDM) for developing instruction for the web. Cognitive flexibility theory focuses 
on the nature of learning in complex and ill-structured domains. Learning is best achieved 
within this environment through the use of non-linear and multidimensional exploration 
of subject matter (McManus, 1996). Hypertext, by its nature, is conducive to supporting 
the presentation of information from multiple perspectives to promote learning in 
accordance with cognitive flexibility theory. 

HDM (Figure 2.2) employs a “geography/cartography design metaphor.” 
(McManus, 1996) A design metaphor is primarily concerned with how the designer 
structures the learning domain while creating the learning environment. Unlike traditional 


instructional systems development, HDM focuses on learner objectives—“what the learner 
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actually comes to the environment wanting to learn,” (McManus, 1996) and provides the 


guidance to help the learner meet those objectives. 


Define Your Identify Cases within 
Learning Domain the Domain 


Identify Themes 


& Perspectives 
in Provide Learner 


Controlled Navigation 
Through Cases 
Map Multiple 
Paths Linking Cases 


Focus Learner 
Self-Reflection 


Figure 2.2 Hypermedia Design Model (McManus, 1996) 





Gagne (1979), Gery (1987) and Alessi (1991) have formulated other systematic 
design steps. Easton (1997) best summarizes the approach to be taken when considering 
developing CBT courseware- “it must be better thought out than more conventional 


training strategies.” 


D. AUTHORING TOOLS 


The final area that warrants discussion is that of CBT authoring tools. Older DOS- 


based authoring systems such as TenCORE, Quest and Phoenix Micro have lost a sizable 
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portion of their market share to newer Windows-based systems. The proliferation of 
training programs which incorporate multimedia; coupled with the dominance of 
Microsoft Windows in the workplace, has reshaped the CBT authoring field (Fritz, 1993). 
The current trend in CBT authoring software development is to provide products that 
incorporate tools for course design, course delivery, web integration and course 
administration features, in one package. 

The traditional CBT authoring process involved an instructional designer who 
knew what the program should do and a programmer who knew the authoring language. 
The evolution of easy to use authoring tools has enabled both instructional designers and 
non-instructional designers to create CBT courseware without the use of a programmer. 
This technological advancement also harbors the potential conseguence of poorly 
designed courseware being produced by non-gualified people. (Hefner, 1996) 

In The 1996 CBT Report (Kemske, 1996), survey respondents identified 139 
different CBT or multimedia authoring products that they either use or have used 
previously. According to the report findings, the five most commonly used authoring 


products were: 


1. Authorware 
2. IconAuthor 
3. ToolBook 

4. Ouest 


5. Director 


28 


The report results disregard software version and implementation (DOS or 
Windows). Author Ware, IconAuthor, ToolBook and Quest are considered dedicated CBT 
development products. Director is ordinarily considered to be a presentation developer 
(Kemske, 1996). Table 2.7 provides a comparison of the current Windows version of 


these products. 


3.5 11 Director 6.0 
N A __ Instructor i Studio 
| Authoring | Icon/flowchart | Icon/flowchart | Page/object | Icon/flowchart, Icon/object 
authoring 
Capabilities 
Wizards 
Programming OpenScript ANSI C Lingo, Xtras, 


Language C++, Java 
Capabilities 


| Hypertext Yes Yes 
| Capabilities 
























Xcommands, 





Yes 





*Hyperlinks can be established by using Lingo scripting 


Table 2.7 Latest CBT Authoring Tool Comparison 
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III. METHODOLOGY 


In this project, the Instructional Systems Development (ISD) model, introduced in 
Chapter 2, was used as the framework for the structured design and development of the 
CBT program. ISD is an iterative process in that each phase provides feedback to one or 
more previous phases for the purposes of modifications and enhancements. The major 


phases involved in ISD are: 


e Analysis 

e Design 

e Development 

e Implementation 


e Evaluation 


The purpose of this chapter is to describe the process involved in applying the ISD 


model to the Electre training presentation. 


A. ANALYSIS 


1. Determine Training Technigue 


The concept for the training was extremely unigue in that the goal was to combine 
hypertext documents, multimedia-based business case studies and an on-line software 
application, through seamless integration. The resultant product would be delivered over 
the World Wide Web utilizing Internet protocols. The multimedia-based case study is the 


main portion of the training program. The case study is a simplified version of an actual 
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application of the Electre method to solve a decislon support problem. As a result, the 


demonstration/presentation technique for CBT program design was selected. 


2. Determine Training Objectives 


The program was designed to supplement classroom instruction and/or textbook 
reading on the Electre method. The desired outcomes were to: 
e Introduce the student to the Electre method via creation of a hypertext 
document of the corresponding textbook chapter 


e Present the student with an animated demonstration of the Electre 
method’s application to reinforce knowledge gained from the reading 


e Enhance student learning and retention through a self-testing module 


e Provide the student with understanding of the underlying methodology 
of the Electre portion of the Co-oP decision support software program 


B. DESIGN 


1. Script Design 


The script was created utilizing the facts contained in the case study. Additional 
characters were created in order to develop a dialogue that would stimulate the user's 
interest. Careful consideration was given to designing interactions that would not 
intimidate, frustrate or insult the user. 

The teach-back approach was used to derive initial ideas for the script. Questions 
and explanations arising from the live presentation of the Electre method were used in the 


actual script. Beta testing was performed using the first draft of the script to present the 
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Electre method to several volunteers. Volunteer guestions and feedback were used to 


modify the script's content. The final script is included as Appendix A. 


2. System Considerations 


The minimum system configuration was one of the first design considerations 
discussed. This configuration had to be such that a majority of computers would be able 
to play the training module. To this end the following is considered to be the minimum 


system configuration to use the training module: 


e 486 compatible CPU 

e At least 30 Mb free hard disk space 

e Super VGA video card with a monitor capable of displaying 640x480 
e Multimedia capable sound system with speakers ` 

e Mouse 


e Internet Explorer with the Shockwave plug-in. (Shockwave can be 
downloaded from the Co-oP Web site) 


e Access to the Internet (recommend a 28.8 bps modem or higher) 


A faster CPU will improve playback of the video. The limiting factor for viewing 
this training module over the Internet is the speed of the connection, which affects the 


download time. 
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3. Media Selection 


Both digital motion video and animation were considered as media options for 
developing the case study presentation. Each format has its own strengths and 
weaknesses. The evaluation was based on choosing the media that would create a guality 


product given the computer resource limitations of both the developer and end-user. 


a. Motion Video 

Motion video will produce motion picture like quality on a computer. 
Professional results can be achieved with minimum investment. To get the video from the 
tape to the computer requires a video capture card. The average cost for a card that 
produces a quality picture is somewhere in the $1,000 range. 

The resolution, frames-per-second (fps), color depth and duration (in 
seconds) are needed in order to calculate storage requirements for full motion video. The 
resulting value is the storage requirements in bits. To convert this value to bytes, divide 
by eight. The following is an example for uncompressed video: 


Video Duration: 5 minutes. 


Resolution: 640 x 480 with 24-bit color 
Frame Rate: 16 fps 
Compression: uncompressed 


Storage space required = (5 (minutes) x 60 (seconds) x 640 x 480 x 24 x 
16)/8 (bits per byte) = 4.5 gigabytes [Using a compression technique like 
MPEG or motion JPEG can reduce this to 1.5 megabits per second 
(Andleigh & Thakrar, 1996)] 
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Full motion video clips can still be very large even when a compression 
scheme is used. To play MPEG or motion JPEG video files on a computer may reguire a 
video card that is capable of decoding these file formats. 

Full motion video provides fluid playback with professional results 


however, this format was not chosen for the following reasons: 


e the large file size of the clips 
e the inability to control the clips within the training program 


e the potential requirement for a higher quality video card in the 
playback computer to decode the video clips 


b. Animation 


Animation is the colorful illustration of movement with rich sound. 
Animated imagery, unlike video, can be made machine readable (Hapgood, 1997). As a 
result, animation requires less bandwidth and storage space. Objects can be created that 
exist only in the mind of the creator and manipulated in any manner. 

Animating bitmaps allows for greater developer control over file sizes. 
There are numerous collections of bit-mapped images available commercially as well as 
on the Internet. Images may be selected from any of these collections. The copyright 
information describes license requirements and limitations for use of the images. 

The use of a bit-map editor is required to modify a single image in order to 
produce other images that can also be used in a production. This process of cutting and 


pasting a single image to produce several others can be very time consuming. This is 
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partly why it takes approximately 200 man-hours to complete one hour of finished 
multimedia (Kemske, 1996). 

Animation file size is directly related to bitmap resolution, measured in 
pixels per inch, and color depth. Resolution and color depth impact upon the guality of 
the bitmap image. As discussed previously, animation reguires less storage space then 
video. Part of the reason for this is that the bit-mapping process can be partially done by 
the CPU of the computer the images are being shown on. Although this may reguire less 


storage space, it will also reguire more computational power from the CPU. 


4. Audio Considerations 


Sound is an analog signal that is normally converted to a digital signal for use by 
the computer. One way to convert this signal is to take many small pictures of the analog 
signal’s waveform. Each of these small pictures is called a sample. Sampling rate is the 
number of pictures that are taken in a second. The actual process of converting an analog 
signal to digital is much more complicated and beyond the scope of this discussion. 

Sound quality is measured in terms of the sampling rate at which the sound is 
recorded. A sampling rate of 44.1 KHz indicates that 44,100 pictures per second are 
taken of the analog waveform. A sampling rate of 4 KHz at 8 bits is considered the 
minimum acceptable for (mono) voice-grade sound. A sampling rate of 8 KHz at 16 bits 
is the minimum required for music. A CD is sampled at 44 KHz at 16 bits. 

The higher the sampling rate, the better the sound quality. The number of bits 


used to store each same also affects the sound quality. 8 bits and 16 bits are commonly 
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used. The more bits used, the better the sound guality. The trade-off is that if a higher 
sampling rate or a higher number of bits is used, the size of the file increases 


dramatically. 


5. Software Considerations 


Several software packages for creating a CBT were considered. An authoring 
package, a graphics program, and audio software were reguired to create the training 


module. 


a. Authoring Packages 


In selecting an authoring system, consideration was given to each of the 


following: 


e the cost of the system 
e training that may be required to learn the system 


e whether the system provided the required functionality 


There are several types of authoring systems. Mastering an authoring 
system can become very time consuming. A dedicated authoring system is the simplest 
type of system. It is designed for a single user and generally only allows for single 
streams or a single object. Timeline-based authoring systems arrange objects along a 
timeline. The timeline is on the screen or in a window. The user selects an object and 


must position it in the timeline. Structured multimedia authoring is based on object-level 
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construction. It is designed for very complex multimedia presentations. It allows for 


modular construction. 


b. Graphics Programs 


The graphics program should allow for the manipulation of different 
image formats. The program’s features should allow the user to cut and paste bit-mapped 
images and then create new images using this cut and paste technique. The program 
should have the ability to save these images to any of the popular file formats. 

Image conversion is also an important consideration. A flexible program 
allows for converting to different file formats and changing the size and/or color scheme 


of the image. 


c. Audio Software 


The software selected should allow the user to record, convert, and edit 
sound files. These features are usually available in one program. For recording, the user 
should have the ability to select different input sources, i.e. microphone, CD, and video. 
A graphical display that shows the waveforms can be used to edit audio files. The ability 


to cut and paste the sound files is also important. 


C. DEVELOPMENT 


A timeline-based authoring tool from Macromedia called Director Suite was 


selected for developing the training module. The initial version was 5.0 and was later 
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upgraded to 6.0. Director was selected because it has an intuitive interface and appeared 
easy to learn. The software interface closely resembles a Hollywood movie production, in 
that both contain cast members, scripts, score and stage. Bundled with the Director suite 
were several other programs for sound editing, 3-D rendering and some very good 
graphics tools. 

Several decisions had to be made regarding format for the movie. Animation 
rather then video was used due to the size of the files related to video. Download times 
for the movie also contributed to the choice of animation. Screen size was set to 640x480 
to ensure that any user could view the movie from the Internet regardless of their screen 
size. Color depth was set to 8-bit to allow for compatibility with older video displays. 
Audio settings were set to a 22KHz sampling rate at 16 bit to provide high guality sound. 


These settings gave the best sound guality while limiting reguired storage space. 


1. Animation Creation 


As discussed previously, animation was chosen rather than motion video for use 
in the training movie due to the reguirement to make the training available over the 
Internet. An exhaustive search was conducted through several collections of bit mapped 
images to select those images that could be manipulated to obtain the necessary 
characters and props for the animation. 

Resolution and color depth were modified to control file size. These trade-offs 
were done for those portions of the presentation where high resolution and color depth 


were not integral to effectively presenting the information. Careful consideration was 
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given to balancing trade-offs in order to avoid degrading the guality of the overall 
presentation. 

Various graphics programs were used to make alterations to the images chosen 
including Paint Shop Pro, Microsoft Paint, Freelance, Director XRes, and the Director 6.0 
paint window. Paint Shop Pro provided a good all around program allowing for importing 
art from many different types of formats, making alterations as needed, and storing the 
altered picture in a useful format, such as a BITMAP file (.bmp), for importing into 
Director 6.0. The paint window within Director 6.0 also provided a useful tool for making 
alterations to pictures once they were imported into the movie’s file. 

Using these graphics programs to cut and paste the images produced several 
images different from the original. These images were then stored as new images. This 
was a very time consuming process. The cutting and pasting of a single image to produce 
several other images took several hours. 

The bulk of the animation’s creation was accomplished within Director 6.0. The 
score within Director 6.0 provided the ability to put these different images together 
thereby giving the appearance of motion. Ten seconds of animation took approximately 4 
to 6 hours of production time. 

Director handles each object and sound within the movie as movie cast members. 
A movie score is used by Director to make each cast member, both visual and sound, 
appear at a given time and place within the movie. Control over when and how each cast 


member enters and exits the movie is also controlled in the score with a variety of 
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transition effects. Many different transitions were used to break up the flow of the movie 
and provide some variety for the viewer. 

Control over the movies’ flow and execution is also provided in Director's 
scripting language called Lingo. Lingo provides the ability to control many of the movies’ 
features such as navigation, play back delays, and interactivity commands. Lingo was 
used to contro] navigation, control sound play back levels, and provide user feedback 


within the training movie. Some sample Lingo script is included in Appendix B. 


2. Audio Creation 


Audio in the training module includes both voice and music. The WAVE (.wav) 
audio file format was used for recording due to its universal use and the flexibility 
provided within its many optional sampling rates. Various sampling rates were tried with 
predictable results. Initially, an 8-bit, 11KHZ audio test recording was conducted. The 
result was a smaller file size with poor sound guality due to a high noise level. A 
compromise between providing high quality sound and producing the lowest media file 
size was reached by recording the sounds at a rate of 22KHz. 

Two computer-recording programs were used for the recordings: Sound Blaster's 
Wave Studio and Sound Forge. Wave Studio was easier to use, but Sound Forge provided 
a greater choice of sound editing and manipulating features; many of which were not 
needed for this project. 

Individual recordings of each statement made by the movie's participants were 


made. This allowed for more precise control and synchronization of cast members in the 
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Director 6.0 score. Synchronizing the player's voices with the movie animation was a 
time consuming, interactive process. Introducing sounds into the movie in short segments 
rather than one long dialog facilitated the job of synchronizing the action with the sounds. 
This does not present a problem as long as the visual cast members are processed to 
appear along with the sound cast members. Delays were utilized within the score to 
synchronize audio and animation. Synchronization was assured throughout the movie by 
breaking the movie up into logical production segments. 

Director 6.0 does not allow for stopping or pausing a sound file during playback 
and then continuing or resuming the sound file from that point. The animated portion of 
the movie can be paused and restarted, but the sound file associated with that portion of 
the movie must restart at the beginning of that sound cast member. This means that in 
order to stop the movie and then resume playing the movie, reguires that both the 


animated and audio portions of a given movie segment be replayed. 


3. Movie Layout 


The movie was divided into two parts during development. This was done to 
facilitate movie production and ease of Internet access and downloading. Part I introduces 
the movie and case study. Part II demonstrates the ELECTRE 1 method of GDSS. Color 
selection within the entire movie was based upon the developers* preferences. 

Part I uses animated cartoon type graphics on a photo background as well as photo 
images on an animated background to demonstrate the usefulness of both animated and 


photo images. The training was introduced in an office environment. 
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Part II presents the training material on a simulated computer screen to represent 
the use of computers in GDSS equipped conference rooms. The screen was divided into 
four areas to emphasize particular areas of the training materials. The four areas from top 
to bottom are: title area, working area, background information area, and navigation area. 

The title area provides a title for the given section of the movie being played. The 
working area 1s centrally located on the screen and contains the critical information for 
the training topic under discussion. Bolded text is used to emphasize the current 
information being presented by the training tool. Transitions are also used to draw the 
viewer's attention to a particular part of the screen and to keep the viewer engaged. 

The background information area is used to provide the viewer with a quick 
reference to previously displayed information that is needed for input to the current topic. 
Bold text is used to highlight pertinent information for the current discussion or 
calculation. Transitions are used to focus the viewer’s attention. 

Navigation tools are placed at the bottom of the computer to facilitate movie 
viewing and navigation. Navigation buttons provided allow a user to move between 
sequential areas of the movie and to the Main Menu or Exam section. The user is notified 
by animated graphics when the current training section has been completed and it is time 


to move on to the next section. Figure 3.1 is a sample screen from Part II of the movie. 
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Figure 3.1 Sample Screen 


D. IMPLEMENTATION 


The next step in the ISD model required that each phase’s decisions up to this 
point be bought together in a final training product. The overarching goal for this training 
module was to make it available to users on the Internet. A Web-based integration 
scheme made up of text-rich Web pages hyperlinked together was chosen for the training 
module. This design scheme provided a logical base from which to present other 
graphical and animated training media. The training package was put together using some 


of the most up-to-date tools available for Web site design and implementation. 
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La Web Page Implementation 


Web pages are made up of HyperText Markup Language (HTML) and can present 
many different types of media to the user including text, graphics, sound, animation, and 
video. Most major vendors of word processing software are beginning to include the 
capability to save and convert files into HTML format. Desktop publishing programs 
such as Adobe’s PageMaker 6.5 and Microsoft’s Word 97 have this ability. 

NetObject’s NetObject Fusion 2.0 was chosen to build the site's Web pages. This 
program provides the capability to create an entire Web site with automatic links between 
each layer of pages. Objects can be imported and placed within any Web page desired 
ranging from text, which can be typed in many popular word processing programs, to 
graphics, animation, sound, and video. NetObjects automatically creates the HTML code 
as each object or string of text is inserted on the Web page. | 

The creation of frames and the placement of navigation buttons and text within 
those frames provide navigation within the Web site. This design gives a visual clue to 
tie user that other areas exist at the Web site and provide a ready means to return to parts 
of the site previously visited. Links are also placed within the main body of each page to 
allow moving to definitions and training media contained on other pages. NetObjects 
provides a guick method for creating and designating each of these types of links. 

There are also a large number oi Web site layout schemes within NetObjects 
including buttons, background colors, and text formats and colors. Simple, subdued 


colors with a large use of black text on white background were chosen to provide for a 
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more universal look and to make the text easy to read. Choosing simple colors also works 


well with different browsers and various display settings. 


2 Movie Internet Preparation 


Director 6.0 prepares a movie to play on the Internet by compressing it and 
changing its format to a .dcr movie. Director 5.0 used a feature called Afterburner to 
"Shock" the movie for the Web. Director 6.0 provides a save option in the file menu to 
save the movie as a Shockwave Movie. An option is also available to create a stand alone 
executable movie called a projector. A great deal of computer processing is required 
during this compression evolution which can be time consuming and even overpowering 
when large movies are processed on computers with slower CPUs or limited hard disk 
space. 

Most of the development work was accomplished on a Pentium 166 computer 
with a 1.6 gigabyte hard disk and approximately 60 MB of free space. The first portion of 
the training movie required 32 MB of disk storage as a Director movie but only 3.4 MB 
as a compressed Shockwave movie. Processing the compression on this segment did not 
cause any noticeable problems with the computer. However, the second portion of the 
training movie was considerably longer requiring nearly 90 MB of storage space prior to 
compression. 

A problem was encountered while Director was compressing the second portion 
of the movie. As director goes through the process of compression it must first expand the 


movie. In the case of part two, the movie expanded to 158 MB which required more 
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space than the hard drive had available. It also took a great deal of time going through the 
process. The final processing on this portion of the film was accomplished on a computer 
with a Pentium 233 processor and 4 gigabyte Ultra-SCSI hard drive, which significantly 
decreased the compression processing evolution. The hard drive had in excess of SOO MB 
of free space, which supported the processing requirements of Director. The final size of 
the second part after compression was 4.1 MB, a significant reduction making it Internet 


ready. 


E. EVALUATION 


The final phase of the project development cycle involved the use of a survey to 
gain valuable feedback from the program users regarding the training application. The 
feedback areas of interest for the development team were the levels of knowledge gained 
by the user, the user’s opinion of the Web site and graphics, and the type of computer and 
Internet connection equipment used. The NetObjects Fusion program was used to create a 
Web interface with a database to capture the users’ answers to the survey. A flat database 
was created with Microsoft Access to capture the inputs gained from the Web based 


survey. Appendix C contains a printed version of the Web-based questionnaire. 
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IV. OPERATION OF THE GROUP DECISION SUPPORT SYSTEM TRAINING 
MODULE 


A. INTRODUCTION 


The major emphasis of this thesis is to create computer-based training (CBT) for a 
decision support system model and to make the CBT available for access via the Internet. 
The ELECTRE 1 multi-criteria decision support model and the Co-oP decision support 
system application were chosen for use to demonstrate the feasibility of this application 
development as well as to identify areas of difficulty and concern when applying CBT to 
an Internet based training (IBT) environment. 

This chapter will discuss the IBT application, procedures for running the IBT, and 


Internet browser requirements. 


B. INTERNET-BASED TRAINING APPLICATION 


The IBT application was developed as one module embodied of two parts: 

e The first part consists of web pages, which provide hyperlinked text-based 
information about numerous models used in multi-criteria decision making. A 
more detailed discussion is provided for the ELECTRE 1 model, which is the 
focus subject of the CBT module 


e The second part is a case study involving a decision-maker, two assistants for 
the decision making process, and an actual business decision to be made 


The web page documents provide an introduction to all of the decision support 
models that are used by the Co-oP DSS. Hyperlinks for key ideas and definitions are 
provided to assist students in their learning and understanding. The ELECTRE 1 model 


pages have been enhanced with excerpts from Co-oP: A Group Decision Support System 
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for Cooperative Multiple Criteria Group Decision Making (Bui,1987), to provide the 


student with a thorough explanation of this model prior to viewing the case study portion 
of the IBT. 

The second part of the IBT is an animated, interactive training movie that has 
three sub-lessons. These consist of an introduction, the ELECTRE 1 model application 
example, and an exam. The presentation begins by providing a list of course objectives, 
an introduction to the decision-makers, and the case study details. Figure 4.1 is a screen 
shot of the introduction, which provides the course objectives for the ELECTRE 1 
training module. These objectives provide the student with the overall course outline, 
which is followed throughout the animated portion of the training. The student will have 
a good understanding of the material if each of these points is understood, and will be 


able to demonstrate this knowledge at the end of the training portion of the module. 
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ELECTRE Training Module Enabling Objectives 


Understand the ELECTRE Method as it applies to 
Decision Support Systems to include: 


Be able to identify decision options. 


Be able to define evaluation criteria and assien 
-. weights to those criteria. 


Be able to create a grading system for evaluation 
the different options, 


Be able to understand and create Concerdance 
and Discordance Matrices. 


Be able to understand and apply the concept of 
Thresholds to the Concordance and Discordance - 
Matrices to create the Outranking Matrix. 


Be able to use the Outranking Matrix to select the 
MET GT | 





Figure 4.1 Introduction/Course Objectives 

The second sub-lesson presents the ELECTRE 1 model in detail with illustrations 
of its application in the case study. The student 1s also provided examples of this case 
study's application in the Co-oP GDSS with screen shots and explanations of how to use 
the Co-oP program. 

Figure 4.2 is a screen shot of the opening screen of the ELECTRE | training 
section that breaks the training down into separate lesson subjects. This menu allows the 
student to navigate within the training module section's training subjects, to go to the 
exam section, and to exit the module when complete. Each of the listed sections can be 
viewed successively while proceeding through the training. They can be accessed directly 


from the menu, which supports the review of a particular section if the student desires. 
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«> ELECTRE m 


TT TT TS AT TRT es 


Part 2 Main Menu: ELECTRE 1 Method Explained | 


A ne ee EE aaa aaa aaa aaa aaa a aaa 


Define Options {Alternatives} 


Deñne Selection Criteria 


Define Weight of Criteria 


Define Grading and Evaluation of Options 


Define Concordance Matrix 


Define Discordance Matrix 


Determine Final Decision 





GEREI EE 


LAAN DEE DON LN N ONA MO DNA POS AN ER 





Figure 4.2 Menu of Part II 


The third sub-lesson tests the students level of understanding of key points 
presented in the training module and provides for revisiting applicable training areas 
within the IBT. It contains both theoretical and practical guestions taken from portions of 
the web page training information and the animated movie. Positive and negative 
feedback are provided to guide the student to the correct answer. The student may also 
return to the training movie’s ELECTRE 1 sub-lesson to review any of the key areas 


presented in order to refresh and solidify his or her knowledge. 
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Cs PROCEDURES FOR ACCESSING AND RUNNING THE IBT 


This GDSS training module can be run by accessing the Co-oP training Web site, 
which is located at [http://www.cimnet.nps.navy.mil/coop]. The Web site training is 
enhanced due to the availability of Web pages containing textual instruction in the 
various DSS models with a focus on the ELECTRE 1 method. 

A Web browser such as Netscape or Internet Explorer must be used to access the 
Co-oP training site on the Internet. Web browser usage and restrictions for this 
application are discussed in the next section. Instructions regarding site content and 
navigation are provided on the Co-oP Home Page with links to Macromedia’s Shockwave 
plug-in to enable the students to download the plug-in. Details on obtaining the plug-in 
are also discussed in the next section. 

Access to the ELECTRE 1 animated training movie is provided from the 
ELECTRE 1 Web page. The movie is built in two parts, which was designed to shorten 
the lengthy down load evolution for the entire movie. A table listing approximate Internet 
download times is provided. 

Part I presents an introduction to the movie and explains the navigation buttons 
used throughout the movie. Figure 4.3 is a screen shot of the introduction and figure 4.4 
is a screen shot of the navigation scheme used throughout the movie. Unlike other CBT 
applications, this Shockwave movie does not allow for synchronous playing of the 
animation and the audio portions. Therefore, the stop button can be used to stop at any 


given spot in the movie, but the movie cannot be restarted from that position. The current 
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section being played must be started over in order to provide for the built in 


synchronization between the animation and the audio. 


sr COOP TRAINING 1 | TT 


elcome to the ELECTRE training movie 


This movie is made up of two parts; 
Part 1: Introduction 
Part 2: ELECTRE 1 
Click on the arrow below to start Part I. Part 2 wil 
played following the completion of Part 1. 





Figure 4.3 Introduction 
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= COOP TRAINING 1 


Navigating within the Movie: 


Click on this 
button to 
Start the Movie 


LA od 





Figure 4.4 Navigation 
A link to Part II of the movie is provided within the Part I section as shown in 
Figure 4.5. The student is also provided a means to exit the training at any time within the 


movie. 
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<7 COOP TRAINING 1 


A = ed: 


Click on Button to Replay Part 1| 


Click on Button to Play Part 2 


Wwe En ea u: 
RN - me te S 
+ a j = AS ORI de ER Ly 
ER t Se. AE ar x 
A y ES 
4 i AREN Ze 
A 9% 
A 


$ 





Figure 4.5 Play Part II 
An exam is provided for the student following completion of the ELECTRE 1 
Web page and movie training. Figure 4.6 is a screen shot of a sample question with the 
instructions for taking the exam. The picture also shows the navigation available to return 


to Part II 5 Main Menu in order to go to any lesson section for review. 
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«¿> CODP TRAINING 1 


e 


OS | 

vere res ee tee PCIE EAE BE AA roete vr pepper de Re RE EP LA Se Ere Pe ee oe ER Re PP Pe PA Ee oe eed DILDO LALLA FI}: 
Use the "Next" arrow to go to the next question. 
Use a mouse to select the correct answer. 


Return to the Main Menu to review a question's topic. 


: Question ELECTRE is characterized by. (source: excerpt from Bui, 
101 S.. Lecture Notes ın Computer Science - Co-oA 


SOLE ORONO NOOR NATHAN’ 


SAJAND 


A. Circumventing the problem of incomplete comparability 
of alternatives through its concept of outranking. 


B. Deciding between non-dominated alternatives: e.g., 
option AI is better than A2 for some criteria and A2 is 
better than Al for some other criteria 


A 
pa 


C. Ill-structuredness and occasional inconsistency of the 
decision maker's preferences. 


po. 
: 
i 
| 
; 
; 
% 
$ 
E 
$ 
í 
E 
E 
3 


NENA DAA 


D. All of the above. 





Figure 4.6 Exam 
Once a student peruses the Web site and interacts with the ELECTRE | training 
movie, a questionnaire included in Appendix C is provided to allow the student to give 
feedback on the training site. This questionnaire was designed to obtain student input on 
the quality of the training, quality of the Web site; opinions on the feasibility of Web 
based training, and user’s platform configuration. The results of the survey are captured in 
a database, which can be used to do statistical analysis on the data provided and to 


identify areas needing future improvements. 


D. INTERNET BROWSER REQUIREMENTS 


The use of multimedia within Web documents is becoming a major focal point of 


Web development. This is because the Web is a natural medium for the distribution of 
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media-based information resources across computing platforms. In addition, creating 
Web ready media resources has become simpler as development and conversion tools 
continue to evolve. 

There are a number of things that must be considered prior to downloading the 
animated ELECTRE 1 movie. They include the type of Web browser being used, the 
computer's processor, the connection type in use for the Internet, and the status of the 
hard disc drive on which the Browser files are cached or stored. 

Multimedia materials referenced on Web documents cannot be played back within 
the Web browser. Instead, when a browser encounters a media format it can not display, it 
looks for a helper application appropriate for the format. Newer Web browsers are now 
being developed with many of these helper applications built in, however, adding media 
viewers as plug-ins can support most media formats. 

To view the multimedia-animated movie provided with this IBT application, the 
free Shockwave plug-in from Macromedia [http://www.macromedia.com] must be 
installed in the student’s Web browser. 

The two most popular Web browsers in use today are Netscape and Internet 
Explorer. The ELECTRE 1 training movie has been tested on both the Netscape 3.01 and 
Internet Explorer 3.0 browsers with the latest Shockwave plug-in installed. Though both 
of these browsers support the Shockwave format, we have found that the movie does not 
play properly on Netscape due to a restriction within the Naval Postgraduate School’s 


Web server software. The movie will play properly through Internet Explorer 3.0 with 
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the most recent Shockwave plug-in installed. This information has been included on the 
appropriate Web pages in the training module. 

Movie downloading has been tested on various computer platforms and types of 
connections. The computer platforms ranged in processor type from Pentium 166s to 
Pentiumll 233s. Very little variance was noted in the processing of this movie between 
the different processor types. Students using these ranges of computers will not notice any 
significant difference in processing speed. 

Movie downloading was also tested on various connection types ranging from 
28.8 Modems to T-1 line connections. A significant difference in download times was 
expected and noted between these various types. The connection type rather than the 
computer's processor is by far the factor with the greatest impact on the movie’s download 


time. Table 4.1 shows the approximate download times for a variety of connection types. 


Type of Connection Part 1 Part 2 
| 3.4 Mb 4.1 Mb 
(1.5 Mbps) 
Cu 
(10Mbps shared) 


Table 4.1 Download Times 
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Web browsers use temporary storage space on the computer's hard disc drive to 
store Web browser downloaded files. In the case of Shockwave movies, the hard disc 
space is also used for temporary storage while the compressed files being received from 
the Web are processed in order to play back on the computer. Normal browsing of Web 
pages does not have a significant impact on the hard disk's storage, but multimedia of the 
size of these movies, reguires a significant amount of additional hard disc space to load, 
process and run properly. It was found that a hard drive with less than 30 MB of free 
space had a significant negative impact on downloading and playing the entire ELECTRE 
| training movie. Having a minimum of 30 MB available free space on the browser’s hard 
drive prior to attempting to view the ELECTRE 1 movie is recommended. This space 
should be available on the drive in which the browser stores its downloaded pages, which 


is normally the C: drive. 
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V. SUMMARY AND RECOMMENDATIONS FOR FUTURE RESEARCH 


A. SUMMARY OF RESEARCH 


This thesis was initiated to research the feasibility of using various Internet 
technologies to provide computer-based training for a Group Decision Support System 
application. Current trends in Internet development and a few of the most up-to-date 
development tools were examined. This research supports the idea of using the Internet as 
a viable means for providing training material to a broadly distributed audience. It also 
demonstrates the usefulness of some of the latest tools provided for this type of 
application development. 

The IBT method of distributing and executing training material has some 
advantages that can be useful to the military. This method provides the training material 
to an almost limitless audience. The only requirement for a user to access the material 
would be to have an Internet connection and current browser. The training material’s 
controlling organization could make changes and updates to the material and almost 
immediately provide the updated material for use by the next user/student with a 
minimum cost for distribution. 

The limiting factor to full usefulness of the Internet for distributing training 
materials is still the bandwidth available to the user. Most modern CBT modules contain 
animation and/or video and sound typical of the ELECTRE 1 movie. Use of this type of 
media results in large file sizes even when the most up-to-date compression techniques 


are used. Downloading these training movies from the Internet through a modem, which 
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is the most probable type of connection for most users, takes a significant amount of time. 
If the computer training material is distributed and used over an organization's Intranet, 
the bandwidth restriction can be overcome making the download and processing time a 
less significant negative factor. 

A modification of the ISD model for training system development was used. This 
design methodology was found to be straightforward in its application and very useful to 
our development team. The following list provides an estimate of the percentage of time 


spent in each phase of the method. 


e Analysis 10% 
e Design 15% 
e Development 60% 
e Implementation 10% 
e Evaluation 5% 
B. LEASONS LEARNED 


The tools available today to create Internet training material provide solid and 
easy to use resources. As inexperienced training and Internet material producers, the tools 
used to develop this project were intuitive and well supported by the software's 
documentation and help function. There are also a number of good Internet resources and 
"How-to" books available by professional users of the tools. 

Using a personal computer to develop multimedia-based training applications for 


use in the Internet requires the latest hardware. A computer with a fast, high powered 
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CPU and a minimum of 32 MB RAM (64 MB RAM if possible) provides the reguired 
horsepower to process movies in the compressed format reguired for the Internet. Use of 
a hard drive with a high access rate and a minimum of two times the movie’s 
uncompressed size in free space, is also recommended to provide the required processing 
support. Use of a high capacity removable storage device, such as a 1 gigabyte JAZ 
storage drive, was also found to be very useful. The JAZ drive provided a back-up source 
and allowed for the transportation of large files to other computer systems. 

The greatest challenge to this development process was creating the animated art 
for the training movie. This aspect of the development phase consumed the greatest 
amount of time and had the steepest learning curve. Based on the experience gained from 
this project’s development, the time required to create this application again would be 
reduced by 25 to 33%. To obtain a more professional animated product will require 
formal training in animation development, or more appropriately, the services of a 


graphic illustrator. 


C. RECOMMENDATIONS FOR FUTURE RESEARCH 


The restriction of download time may be lessened in the future as streaming 
technologies become maturer. Streaming was attempted in this project with unacceptable 
results. As this technology progresses, it should be looked into as a possible enhancement 
to Internet multimedia training materials. 

Future development should be considered for other GDSS models and concepts as 


a result of the successful demonstration in this application of the latest in Internet 
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multimedia development to one aspect of the multi-criteria decision models. Integrating 
Internet training with an Internet ready GDSS application would be a good enhancement 
as well. Migrating the Co-oP GDSS program to JAVA-based applets would provide a 


more versatile resource for that type of application. 
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APPENDIX A. ELECTRE PRESENTATION SCRIPT 


Characters: Narrator 
Mr. Castella 
Mr. Henry 
Mrs. LeFlore 
Applicants 1-5 


Scene 1 
(multimedia presentation has been launched by user clicking link from text) 


Narrator: The following presentation is an example of how the Electre Method was applied to 
an actual problem. The main characters are Mr. Castella (character/obj appears on screen), 
President of Castella Enterprises, Inc., Mr. Henry, (character/obj appears on screen) VP of 
Operations, and Mrs. LeFlore (character/obj appears on screen), head of the Operations 
Research Department. Mr. Castella is searching for a replacement for his head accountant, who 
is retiring. Five applicants (add applicants A1-ASŠ to screen, one by one), answered his 
advertisement placed in the business section of a local newspaper. 


Fade out... 


Narrator: The demo begins with Mr. Henry and Mr. Castella discussing how to make a 
selection from the five applicants. 


Scene 2 


Fade in... Mr. Castella on screen alone looking at applications (papers, folders) 
Mr. Henry appears on screen from left as if entering an office and begins talking 


Mr. Henry: Good Morning Mr. Castella. I understand we have received a total of five 
applicants for the head accountant position. Have you selected the one you wish to hire? 


Mr. Castella: Good Morning Mr. Henry. No, I have not made a final selection however, after 
studying all of the resumes, I eliminated two of the applicants, (dissolve/fade out? A4 and A5), 
whose qualifications were good, but not precisely what I wanted. The three remaining applicants 
(reposition A1-A3 on screen), present a difficult choice; none of them stands out from the 
others as clearly the best choice. I have no idea how best to decide among these three. 


Mr. Henry: Hmmm. (Show Henry with thinking bubble over his head, then change to light 


bulb for idea) (sounding very excited) I have just the answer Mr. Castella! During last week’s 
management meeting, Mr. Andrews in Marketing, briefed the group on how his department 
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applied a decision support model called the Electre Method to determine which campaign 
strategy was best for next guarter's operations. 


Mr. Castella: (sounding doubtful) Electre Method you say? Have you seen a demonstration of 
this model? How does it work? Do you think it will help me? 


Mr. Henry: I have not had the opportunity to see the method demonstrated. Mr. Andrews was 
so enthusiastic about it that I invited him to provide a demonstration for the entire management 
group next month. He was supposed to contact Mrs. LeFlore in Operations Research since she is 
the person who introduced his department to the Electre Method and guided them through the 
steps involved in using it. I think I should contact Mrs. LeFlore and arrange for her to 
demonstrate this Electre Method for you and I using your applicant selection problem. When 
would you like me to set the demonstration up for? 


Mr. Castella: As soon as possible. I have a pretty open schedule for the next two days. See if 
you can make this happen by then. 


Mr. Henry: Very well. I will see what I can do. 
Have Henry heading back off screen form the same direction he entered. 
Fade out... 


Scene 3 
Print on screen “Two days later in the Operations Research Department” 


Fade in.... Mrs. LeFlore is on screen. Mr. Castella and Mr. Henry walk in from the left. 


Mr. Henry: Good morning Mrs. LeFlore (shaking Mrs. LeFlore’s hand). May I introduce Mr. 
Castella. He is very anxious to see the presentation you have prepared for us. 


Mrs. LeFlore: (smiles warmly) Good Morning Mr. Henry. (Turns to Mr. Castella) Good 


morning Mr. Castella. It is a pleasure to meet you. Everything is in place for the presentation 
and I am ready to begin whenever you are. 


Mr. Castella: (smiles, shaking LeFlore’s hand) Good morning and thank-you for preparing 
this on such short notice. I hope what you are about to show us will help me decide which 


applicant is best for the company. 


Mrs. LeFlore: No problem sir. I’m sure we can do something to help you out. If you gentlemen 
will follow me this way, we’ll get started. 


Fade out.... 
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Scene 4 
Fade in... all three characters are in what appears to be a briefing room. Mr. Henry and Mr. 
Castella are sitting and Mrs. LeFlore is standing in front of a large presentation board. 


Mrs. LeFlore: The first part of the process involves data collection. The first step in data 
collection is to define your options (Print on screen “1. Define options” in upper left-hand 
corner), which represent all the different possible actions. We should refer to the three 
remaining applicants by their real names. Mr. Brown is Al (bring/fade-in Al on screen but 
print “Brown”), Mr. Smith is A2 (bring A2 on screen but print “Smith” ) and Mr. Jones is 
A3 (bring A3 on screen but print “Jones””). Do you have any questions so far? 


Mr. Castella: No. Please proceed. 


Mrs. LeFlore: Good. The next step is to define the criteria you would like to use for evaluating 
each of the applicants. Did you bring the papers I had faxed up to your office yesterday? 


Mr. Castella: Yes. I have them right here. I can’t say that I understand everything you asked 
me for. 


Mrs. LeFlore: (Takes the papers Mr. Castella hands her) Thank you. Don’t worry sir. 
Things will all come together as we proceed through the demonstration. 


(Print on screen “2. Definition of criteria” under step 1) 


Mrs. LeFlore: I see you have decided on five criteria Mr.Castella: Education (Print “1. 
Education” to screen), Experience (Print “2. Experience” to screen), Adaptability (Print 
“3.Adaptability” to screen), Sociability (Print “3. Sociability” to screen), and Health (Print 
“4, Health” to screen). Do you still feel these are valid criteria for evaluating an applicant or 
would you like to make some changes” 


Mr. Castella: No, I do not want to make changes. I am satisfied with the criteria already 
chosen. 


Mrs. LeFlore: O.K. The next step is to assign weights to the criteria. (Print on screen 3. 
Weight of criteria” under step 2) If you will recall Mr. Castella, the questionnaire I sent you 
asked you to place different levels of importance on the five criteria with the only condition 
being that the weights had to be divided up out of a total of 100. 


Mr. Castella: Now here is where I started becoming confused. I am not sure that I completed 
this portion correctly. 


Mr. LeFlore: Don’t worry sir, we’ll take a look at what you’ ve written and see if you want to 
change any of your previous choices. Let’s see. According to your answers here, Education is 


assigned 15 (Print “15'' next to “1. Education”). Experience is assigned 15 (Print “15"' next 
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to “2. Experience” forming even column for weights). Adaptability is assigned 20 (Print 
20" in column). Sociability is assigned 30 (Print “30" in weight column). Health is assigned 
20 (Print “20"' in weight column) . The weights you have assigned have to be based on the 
level of importance you feel each criteria holds with regard to your decision making. Would you 
say these values correctly represent your assessment? 


Mr. Castella: Well, now that you have explained it that way, I suppose it makes more 
sense...yes I am happy with the values I have assigned. 


(Print on screen “4. Grading system “‘ under step 3) 

Mr. LeFlore: Very good sir. Now we will help you establish a grading system to use for 
evaluating each applicant. In this step, we will assign maximum numeric values corresponding 
to the following alphanumeric ratings: Excellent, Good, Average, Fair, Weak. After you have 
completed this, the result is a table of points to award for each grade. Do you have any questions 
before we proceed? 


Mr. Castella: No. Not at this point. 


Mrs. LeFlore: Okay then. Please fill out this piece of paper (hands Castella a piece of paper), 
with the scores you would assign to each criteria based on its alphanumeric rating. 


(Show Castella looking over paper and writing on it for a brief period of time before 
handing it back to LeFlore) 


Mrs. LeFlore: Now if we take the numbers you have assigned for each criteria and rating (have 
LeFlore look as if she is reading and either inputting info into a computer or writing on 
presentation board), we come up with the following criteria matrix. (Print on criteria matrix 
on screen. See following table: 


Criteria 
1 2 3 4) 5 
Excellent 10 10 10 20 15 
Good 8 8 8 15 10 
Average 5 5 5 10 5 
Fair 0 1 1 0 3 
Weak 0 0 0 0 1 


Mr. Castella: I think I understand what we’ ve done so far, but where are we headed next? 


Mrs. LeFlore: I’m glad you asked that sir. The next step is for you to evaluate each applicant 
based on the established criteria. (Print to screen ‘5 Evaluation of the applicants” under step 
4) Please fill out this form and base your evaluation on everything you have learned about the 
applicant through resumes, interviews and references. (LeFlore hands another paper to 
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Castella. Castella takes the form and takes some time filling it out. You can have him 
consult some notes in his briefcase or portfolio) 


Mr. Castella: (hands the paper back to LeFlore) I think this should do it. How do we use 
these numbers to come to a final answer? 

Mrs. LeFlore: Well sir, let's have a look at what you've written and then we'll go over the 
procedures for calculating the results. (Print on screen the following matrix: 


Applicant Criteria 

1 2 3 4 5 
Brown 8 8 5 15 10 
Smith 10 1 8 10 15 
Jones 5 10 8 20 10 


Mrs. LeFlore: This is what the scores you’ve assigned for each criteria look like in matrix 
format. Now let’s look at how we calculate the results. 


Fade screen out.... 

Scene 5 
Print to screen “The Results”? 
Fade screen out... 


Print on screen “Step 1 Concordance Matrix Development” in upper left-hand corner of 
the screen. 


Mrs. LeFlore: (speaking to Mr. Castella) The Electre Method uses the concordance and 
discordance coefficients to make paired comparisons between different options. 


Mr. Castella: I’m afraid I don’t quite follow you. What are concordance and discordance 
coefficients and how are they derived? 


Mrs. LeFlore: The Electre Method is based on a rule written by a French philosopher, dating 
back to 1750. To paraphrase, let’s say you have two actions, A&B, which you need to choose 
between. If action A is better than action B for the majority of the decision criteria and there is 
no criteria for which A is clearly worse than B, then it stands to reason that A outranks B. Does 
this make sense to you so far sir? 


Mr. Castella: Yes, I understand what you have said thus far, but this does not explain the 
concordance and discordance you mentioned earlier. 
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Mrs. LeFlore: The concordance and discordance conditions are direct descendants of this 
principle. The condition of concordance reguires that a majority of viewpoints favor A. A 
condition of discordance reguires that no criterion favor B excessively. 


Mr. Castella: (exclaim happily) Now I get what they are, but how do we use them? 


Mrs. LeFlore: The Electre method uses the concordance and discordance coefficients to make 
paired comparisons between different options. In our case, we will use the concordance and 
discordance coefficients to make paired comparisons between Mr.’s Brown, Smith and Jones. 
Would you like me to go over any of this again? 


Mr. Castella: No. I understand everything you’ve said so far. What’s next? 
Mrs. LeFlore: The concordance factor varies from 0 to 1. A perfect value for the concordance 
coefficient is 1. The formula for determining concordance coefficients may be summarized as 
follows: 
Print on screen the following formula: 

Sum of the weights of the criteria 


by which A outranks B 


Sum of the weights of all criteria 
of the model 


Mrs. LeFlore: The concordance coefficient indicates to what extent one option is better than 
another. I will save the discussion of the discordance coefficient calculations for later when we 
are ready to look at the actual values. Is there anything you would like to review before we 
move ahead. 


Mr. Castella: No. Please continue. 


Mrs. LeFlore: Very well then. Utilizing the data in this example, the Index of concordance, 
assuming Mr Brown outranks Mr. Smith, is calculated as follows: 


Print to screen the “Weight of criteria” scale and the “evaluation of applicants” matrix 
Mrs. LeFlore: Performing a pair-wise comparison of Mr. Brown versus Mr. Smith for each 
criteria, we find the following results. Mr. Brown does not outrank Mr. Smith for criteria 1 


(Highlight Brown and Smith and their values under criteria 1 column in matrix) , which is 
education, so a score of zero is assigned. 
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Print on screen the following. Note that we will add values to this calculation as the 
narrator proceeds through discussion of the calculation 


CBrown/Smith= 0+ 


Mr. Castella: Why did you assign a zero? 


Mrs. LeFlore: Whenever option A does not outrank option B for the criteria being examined, a 
score of zero is assigned. 


Mr. Castella: Well what happens when option A does outrank option B? What value do you 
assign in that case? 


Mrs. LeFlore: I’m glad you asked sir. As you will see further on in this evaluation, whenever 
option A outranks option B for a given criteria, the value assigned is the weight of the criteria. 
Let's take a look at criteria 2. Here Mr Brown outranks Mr Smith (Highlight Brown and Smith 
and their values under criteria 2 column in matrix). In this instance, the weight of criteria 2, 
which is 15, is added to the formula. (Print “15 4” on formula screen). Does this make sense 
sir? 


Mr. Castella: Absolutely. 
Mrs. LeFlore: Great. Why don't you do the comparison for criteria 3 sir. 


Mr. Castella: Okay. Looking at criteria 3, we find that Mr. Brown does not outrank Mr. Smith, 
(Highlight Brown and Smith and their values under criteria 3 column in matrix), therefore 
a score of zero is added (Print ‘0 +” to formula screen) How did I do. 


Mrs. LeFlore: Fantastic sir! I think you have the hang of it. How about doing the next criteria? 


Mr. Castella: (smiling and sounding very enthusiastic) For criteria 4, Mr. Brown outranks 
Mr. Smith, (Highlight Brown and Smith and their values under criteria 4 column in 
matrix), therefore a score of 30 which is the weight of criteria 4, is added to the formula. (Print 
“30 +” to formula screen) For criteria 5, Mr. Brown does not outrank Mr. Smith, (Highlight 
Brown and Smith and their values under criteria 5 column in matrix), therefore a score of 
zero is added to the formula. Print “0'' to formula screen. 


Mrs. LeFlore: The outcome of the calculation is .45 or 45%. This value is placed in the 
concordance matrix. 


Print to screen the following matrix which will be filled in as the instructions progress: 
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Concordance Matrix 


Brown Smith Jones 


Brown == 45 
Smith == 
Jones -- 


Mrs. LeFlore: The same steps are followed to determine the Index of Concordance, assuming 
that Mr. Smith outranks Mr. Brown. The results are as follows: 


Print this formula to screen: 


CSmith/Brown= 15+0+20+0+20 


Mrs. LeFlore: This value is added to the matrix in the appropriate column. Print “55” to 
correct column in matrix 


Concordance Matrix 


Brown Smith Jones 


Brown «= 45 
Smith 55 = 
Jones -- 


Mrs. LeFlore: Do you have any questions on what we’ ve covered so far sir? 


Mr. Castella: No. I understand the comparison and calculations. Do we just keep repeating the 
same procedure? 


Mrs. LeFlore: I’m glad this is making sense. Yes, the steps are repeated until all of the 
remaining pairs of alternatives have been evaluated. In an effort to conserve time, the results 
have been filled in for convenience. 


Fill in values for matrix as follows: 


Concordance Matrix 


12 


Brown Smith Jones 


Brown -- 45 35 
Smith 55 -- 55 
Jones 85 65 -- 


Mrs. LeFlore: We will now move on to the discordance coefficient calculations. Recall the 
example from our earlier discussion which required choosing between actions A and B. Well, 
the condition of discordance requires that no criterion favor option B excessively. The 
discordance coefficient indicates to what extent an option contains discordant elements that 
might make the option unsatisfactory. Calculation of the discordance coefficient requires 
unitizing the evaluation scores, not the weight. Just as in the concordance coefficient 
calculations, the discordant factor’s value varies between 0 and 1. A ‘fatal’ score for the 
discordance coefficient is 1. The formula for determining discordance coefficients may be 
summarized as follows: 


Print the following formula on the screen: 


The greatest negative variation 

(i.e., B outranks A) between 

the evaluation scores for a 
single criterion 


DA/B = The maximum range between the 
highest possible score and 
the lowest possible score 


Mr. Castella: This seems rather overwhelming. Perhaps if we move on to the calculations 
things will become clearer for me. 


Mrs. LeFlore: Fine sir. Feel free to stop me at any time if you have questions or would like me 
to go over something again. Once again we will perform pairwise comparisons. Let’s begin by 
assuming Mr. Brown outranks Mr. Smith and examine the evaluation scores they received for 
criteria 1. Print the evaluation scores matrix to the screen. Remember, we are looking for the 
greatest negative variation, so for criteria 1 Mr. Brown received an 8 and Mr. Smith received a 
10. Since we are assuming that Mr. Brown outranks Mr. Smith, he would have to outrank him 
by a negative value of 2. This value is placed in the numerator. Print the following to the 
screen: 


D Brown/Smith = Max (2, 


Mr. Castella: So for criteria 2, Mr. Brown received an 8 and Mr. Smith received a 1. There is 
no negative value here so what would the answer be? 


Mrs. LeFlore: In this instance, we would assign a value of zero. How about trying the next 
one? 
Print to screen: 


D Brown/Smith = Max (2, 0, 


Mr. Castella: For criteria 3, Mr. Brown received a 5 and Mr. Smith received an 8. Therefore we 
would place a 3 in the formula you have on the board. Print to screen: 


D Brown/Smith = Max (2, 0, 3, 


Mrs. LeFlore: Absolutely. To finish off the numerator, criteria 4 would be O and criteria 5 
would be 5. Print to screen: 


D Brown/Smith = Max (2, 0, 3, 0, 5) 


Mrs. LeFlore: Do you have any questions before we move on to the set of values in the 
denominator? 


Mr. Castella: No. Please continue. 
Mrs. LeFlore: Now for the set of values in the denominator, we use the range between the 


highest and lowest scores which could have been assigned for each criteria. Recall our grading 
criteria matrix. Print to screen grading matrix. 


Criteria 
1 2 3 4 5 
Excellent 10 10 10 20 15 
Good 8 8 8 15 10 
Average 5 5 5 10 5 
Fair 0 1 1 0 3 
Weak 0 0 0 0 1 


Mrs. LeFlore: Would you care to take a shot at filling in the appropriate values? 
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Mr. Castella: Sure. Criteria 1 is 10, criteria 2 is 10, criteria 3 is 10, criteria 4 is 20 and criteria 5 
is 14. How did I do? 


Mrs. LeFlore: Perfect. Our formula now looks like this: Print to screen: 


D Brown/Smith = Max (2, 0, 3, 0, 5) 


Max (10, 10, 10, 20, 14) 
Mrs. LeFlore: Now we have to select the highest value from each set. In this case, the 
numerator would be 5 and the denominator is 20. It is important to point out that the 
denominator value will be the same for all of our calculations. The discordance coefficient is .25. 
Print to screen: 


D Brown/Smith = 5/20 = .25 


Mrs. LeFlore: We create a discordance matrix and place this value in the appropriate column. 
Print to screen: 


Discordance Matrix 


Brown Smith Jones 
Brown oe 25 
Smith -- 
Jones EE 


Mrs. LeFlore: Now assume Mr. Smith outranks Mr. Brown. What would our calculation look 
like? 


Mr. Castella: Criteria 1 is 0, criteria 2 is 7, criteria 3 is O, criteria 4 is 5 and criteria 5 is O. 
Print to screen: 


D Smith/Brown = Max (0,7,0,5,0) 


Max (10, 10, 10, 20, 14) 


Mr. Castella: The discordant coefficient is equal to 7 divided by 20 or .35. 
Print to screen: 


D Smith/Brown = 7/20 = .35 


(5 


Discordance Matrix 


Brown Smith Jones 
Brown => AS 
Smith 35 -= 
Jones ” 


Mrs. LeFlore: Fantastic. Now let’s go ahead and fill in the rest of the matrix to save time. 
Print to screen: 
Discordance Matrix 


Brown Smith Jones 
Brown == 25 25 
Smith 35 => 50 
Jones 15 25 -- 
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Mr. Castella: How do we derive a final decision from all of these numbers? 


Mrs. LeFlore: The factors from the matrices are used in conjunction with concordance and 
discordance thresholds. 


Mr. Castella: Where do these thresholds come from? 


Mrs. LeFlore: They are chosen arbitrarily by the user- you in this case. The concordance 
threshold, let’s call is P, varies from 0.5 to 1, and is more severe as it approaches 1. The 
discordance threshold which we will call Q, is more severe as it approaches 0. 


Mr. Castella: Let’s say I decided to make P=.60 and Q=.30. How would I use these values to 
come to a decision? 


Mrs. LeFlore: We create what is called an Outranking Matrix based on an IF-THEN decision 
table which says the following: 
Print to screen while Mrs. LeFlore reads aloud: 


If Then 
CA/B>=P and DA/B<=Q A outranks B 
CB/A>=P and DB/A<=Q B outranks A 
A outranks B, and B outranks A The options are equivalent 
Otherwise The comparison is characterized by 


Indetermination or incomparability 
Mr. Castella: Okay. Let’s see how the matrix is developed. 


Mrs. LeFlore: Well if we look at the situation where we assumed that Mr. Brown outranks Mr. 
Smith, the concordance coefficient value of .45 1s not greater than or equal to our P value of .60. 
The discordance value of .25 is less than the Q value of .30; therefore we cannot say that Mr. 
Brown outranks Mr. Smith. A zero is placed in the appropriate column of our matrix. Print to 
screen: 


Brown Smith Jones 
Brown -- 0 
Smith Es 


Jones -- 


Mr. Castella: How can we conclude this when the Q value was less than .30? 


Mrs. LeFlore: Only one of the two conditions in the IF statement was met. In order for the 
THEN statement to be applicable all conditions in the IF statement must be met. 


Mr. Castella: Okay. That makes sense. Let me see if I can apply this to the case of Mr. Smith 
outranking Mr. Brown. Now, the concordance coefficient value is .55 which is less than the P 
value and the discordance coefficient value is .35 which is greater than the Q value. In this case, 
the IF conditions are not met and a 0 is assigned. How was that? 


Mrs. LeFlore: Great! This is what our matrix looks like so far. 
Print to screen: 


Brown Smith Jones 
Brown == 0 
Smith 0 = 
Jones -- 


Mrs. LeFlore: Let's try one more. How about the case where we assume that Mr. Brown 
outranks Mr. Jones? 


Mr. Castella: The concordance coefficient value is .35 which is less than the P value, and the 
discordance coefficient is .25 which is less than the Q value. Since both conditions of the IF 
statement have not been met, a zero is assigned. 

Print to screen: 


Brown Smith Jones 
Brown -- 0 0 
Smith 0 ne 


Jones a 


Mrs. LeFlore: Just for completeness sake, how about the case where Mr. Jones outranks Mr. 
Brown? 


Mr. Castella: The concordance coefficient is .85 which is greater than the P value and the 
discordance coefficient is .15 which is less than the Q value. Both conditions of the IF statement 
have been met. According to the IF-THEN Decision Table, Mr. Jones does outrank Mr Brown. 
What value do we place in the matrix? 


Mrs. LeFlore: We place a 1 in the matrix. I will go ahead and fill in the rest of the matrix with 
the appropriate values. 
Print to screen: 


Brown Smith Jones 
Brown = 0 0 
Smith 0 =- 0 
Jones 1 1 = 
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Mrs. LeFlore: Now we are ready to draw some final conclusions. According to our table, Mr. 
Jones outranks both Mr. Brown and Mr. Smith, the values of 1 for those conditions support a 
condition of dominance on the part of Mr. Jones. 

Print to screen: (with arrows pointing from Jones to Brown and Smith) 


Brown Smith 


Jones 


Mr. Castella: So, Mr. Jones should be selected for the job, but what about the relationship 
between Mr. Brown and Mr. Smith? Which one is a better choice between those two? 


Mrs. LeFlore: I'm glad you asked. We ended up with a case of incomparability between Mr. 
Brown and Mr. Smith, meaning that we can not prove any relationship between the two. Asa 
result, we cannot draw any conclusions as to the dominance of one over the other. 

Mr. Castella: Isee. (Standing and walking over to shake Mrs. LeFlore’s hand) This has 
been a great learning experience for me. I see now why Mr. Andrews was so excited about all of 
this. Thank you for taking the time to demonstrate the Electre Method for me today. I highly 
endorse your teaching this to all of our Department Managers. 


Mrs. LeFlore: Thank you for your kind words sir. It has been my pleasure. I look forward to 
helping anyone who is interested in applying this method for other decision making situations. 


Mr. Castella: I will have Mr. Henry send out a memo to all the Department Managers. Thank 
you again and have a nice day. 


Mrs. LeFlore: You’re welcome sir. Good-bye (Mr. Castella departs) 


Fade out..... 
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APPENDIX B. LINGO SAMPLES - MACROMEDIA’S PROGRAMMING SCRIPT 
LANGUAGE 


This appendix presents selected examples of Macromedia's script language Lingo. Lingo 
was used to control movie cast member characteristics, navigation, interactivity, and playback 
timing within the ELECTRE 1 training movie. A helpful listing of Lingo commands is provided 
in the Director 6 documentation and in Director 6’s Help files. Examples of Lingo script can also 
be found on the Internet at the Macromedia Web site and links to other sites provided there. 

A. Controlling Cast Member Characteristics 
Sound Cast Members: 
on enter frame - reduces the sound play back level of a sound cast 
set the soundLevel to 1 member when the movie player enters the frame 
end 
B. Navigation 


Moving to specific areas of the when with user input movie: 


on mouseUp - movie player goes to the "welcome" flag within 
go to "welcome" the movie when the left mouse button is "up" after 
end being clicked. 
Movie navigation without user input: 
on exitFrame - keeps the playback of the movie within a frame 
go to the frame indefinitely waiting usually to wait another 
end command 


Navigation to a different movie outside of the current movie: 


on mouseUp - Goes to and starts downloading another movie 
gotoNetMovie "http://www.cimnet.nps.navy.mil/coop/brief2/brief2.dcr" 
end 


C. Interactivity 


Provide feedback to the user given a specific input: 


on mouseUp - Provides the audible reply "clap" to the user for 
puppetSound "clap" the input of mouse up 
end 
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Allow user to control play status of the movie: 


on mouseUp - Stops the movie at the present location waiting 
pause for another input 
end 


D. Control Playback Timing 


Control synchronization by delaying within a frame 


on enterFrame - Stops the visual playback in a given frame 
delay 6 * 60 while the sound channel continues to play the 
end audio member (delay is designated for 6 seconds) 
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APPENDIX C. Co-oP TRAINING SURVEY 


Please complete the following questionnaire after you have viewed the Decision Support System training 
provided at the http://cimnet.nps.navy.mil/coop web site. In particular, we are interested in your 
feedback regarding the ELECTRE 1 movie. Completing the questionnaire should not take more than 15 
minutes. Thank you in advance for your time and interest in providing feedback for this project! 


First Name Last Name E-mail 
(Filling in the personal information fields above is optional) 


1. How much did you know about Decision Support Systems (DSS) prior to the training session? 
O A lot 
O A fair bit 
O A little 
O None 


2. How much did this training session add to your knowledge about DSS? 
O Alot 
O A fair bit 
O A little 
O None 


3. How much did you know about the ELECTRE 1 model? 
O A lot 
O A fair bit 
O A little 
O None 


4. How much did this training session add to your knowledge about the ELECTRE 1 model? 
O A lot 
O A fair bit 
O A little 
O None 
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5. To what degree do you feel this training session explained the concept of Concordance? 
O Alot 
O A fair bit 
O A little 
O None 


6. To what degree do you feel this training session explained the concept of Discordance? 
O A lot 
O A fair bit 
O A little 
O None 


7. To what degree do you feel this training session explained the application of these concepts to the 
process of decision making? 


O Alot 

O A fair bit 
O A little 
O None 


8. Do you feel you will be able to apply this DSS model to a decision-making reguirement as a result of this 
training session? 


O Strongly agree 

O Agree 

O Disagree 

O Strongly Disagree 
O No Opinion 


9. How did you access this training program? 
O Local hard drive 
O CD ROM drive 
O WWW (Internet) 
O Other 


10. The guality of the web site pages is: 
O Very good 
O Good 
O Average 
O Bad 
O Very bad 
O No opinion 
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11. Did you try to download the Director Shockwave movie? 
O Yes 
O No 


12. Could you download it? 
O Yes 
O No 


13. How long did it take you to down load of the movies (Minutes)? 
Part 1: Part 2: 


14. Did the movie play correctly on your machine? 
O Yes 
O No 


15. What is the speed of your Internet connection? 
O 14.4 Modem 
O 28.8 
O 36.6 
O 56 
O 128 
O TI 
O Other 


16. The response time is: 
O Very good 
O Good 
O Satisfactory 
O Unsatisfactory 
O No opinion 
17. Despite some time delay factor, demonstration movies on the Web are of practical use for teaching and 
learning. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


18. Using a Web document is a good and practical way to distribute teaching materials. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 
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19. As shown by this courseware, multimedia makes the learning process better. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 
20. Both sequential access (following the arrows through the tutorial) and branching (via hyperlinks and 
buttons) are allowed in these documents. Do you think that this is a good way to organize a courseware? 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


21. Hyperlinks in the text make it easier to learn. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


22. The courseware is easy to learn and efficient to use. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


23. The courseware is pleasant to use. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


24. It is easy to remember how to use the courseware. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 
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25. The courseware is interactive or potentially interactive. 
O Strongly agree 
O Agree 
O Disagree 
O Strongly Disagree 
O No Opinion 


26. If you are learning/teaching or want to learn/teach DSSs, will you use this courseware or similar 
documents on the web? 


O Yes 
O No 
O Not applicable 


27. The transitions within the movie when played from the Internet may not always be smooth. If you use 
this package as a training aid tool, this is: 


O Quite acceptable 

O Acceptable 

O Unacceptable 

O Definitely unacceptable 


O No opinion 


28. How would you rate the quality of the animations? 
O Very good 
O Good 
O Average 
O Bad 
O Very Bad 


O No opinion 


29. How would you rate the quality of picture and other artwork? 
O Very good 
O Good 
O Average 
O Bad 
O Very bad 
O No opinion 


30. Are you familiar with the World Wide Web (WWW)? 
O Very much 
O Pretty much 
O So so 
O Not really 
O Not at all 
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31. Are you familiar with computers? 
O Very much 
O Pretty much 
O Soso 
O Not really 
O Not at all 


32. Are you familiar with multimedia applications? 
O Very much 
O Pretty much 
O So so 
O Not really 
O Not at all 


33. Are you familiar with computer based training (CBT) applications? 
O Very much 
O Pretty much 
O So so 
O Not really 
O Not at all 


34. How have you accessed and run a multimedia/CBT training course in the past (select all that apply)? 
O CD-ROM (Stand-alone or LAN based) 
O Intranet application 
O Internet application 
O Not applicable 


35. What kind of microprocessor did you use for running this training product? 
O 486-Based Microcomputer 
O 586-Based Microcomputer 
O Pentium-Based Microcomputer 
O Other 
O Dont know 


36. Please specify the amount of system RAM in your microcomputer? 
O 8 Meg RAM 
O 16 Meg RAM 
O 32 Meg RAM 
O More than 32 Meg RAM 
O Dont know 
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37. For my personal training needs, I prefer multimedia/CBT based training for the following reasons 
(select multiple answers if applicable): 


O It is more effective than reading a book 

O It can be replayed many times 

O It can be completed at my own pace 

O It cost less than instructor lead training courses 


O No opinion 
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